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Abstract 

For the measurement of N s signals in N events rigorous confidence bounds on 
the true signal probability p exa ct were established in a classical paper by Clopper and 
Pearson [Biometrica 26, 404 (1934)]. Here, their bounds are generalized to the HEP 
situation where cuts on the data tag signals with probability P s and background data 
with likelihood P& < P s . The Fortran program which, on input of P s , Pb, the number 
of tagged data N Y and the total number of data N, returns the requested confidence 
bounds as well as bounds on the entire cumulative signal distribution function, is 
available on the web. In particular, the method is of interest in connection with the 
statistical analysis part of the ongoing Higgs search at the LEP experiments. 



*This research was partially funded by the Department of Energy under contract DE-FG02-97ER41022. 
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1 Introduction 



The general theory of confidence bounds (or fiducial intervals) was developed by Fisher [[I]], 
Neyman and Pearson ||. We consider a particular problem which is of interests when cuts 
are used to analyze high energy physics data. Typically, a neural network or some other 
method of performing the cuts results in probabilities (efficiencies) to tag signals more likely 
than background events. For instance by means of Monte Carlo (MC) simulations, these 
probabilities can normally be calculated. Let P s be the probability to tag a signal and P b 
be the likelihood to tag a background event, < P b < P s < 1. Out of a total number of iV 
data one gets in this way 

N Y tagged data. (1) 

It is easy to find from this the mean expectation for the signal probability. Assume that 
there are N s signals and N b background events in the data. Then we have 

N = N S + N b and N Y = P s N s + P b N b 

and these two equations solve for 

_ N. _ N Y -P b N 

Pmean- ^ - ^ ^ _ p&) • W 

The question is, what are the implied confidence limits on the signal probability? 

The special case P s = 1 and P b = (sure signal detection) has been treated by Clopper 
and Pearson || in 1934. After briefly reviewing their approach in the next section, we derive 
and illustrate the general case in section 3. This is, in part, based on Ref.Q. In particular, 
the method is valid when the number of tagged data is small and returns the probability Pq 
for the case that there is no signal, i.e. that the exact signal probability is p ex act = 0. This 
is of interest for the statistical analysis of the ongoing Higgs search at LEP |J. Discovery 
of the Higgs particle on the 5<r level would mean P < 0.287 10~ 6 . Subsection |5T5| explains 
the use of the corresponding Fortran programs and how to download them from the web. 
Conclusions follow in the final section 0. 
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2 The Clopper— Pearson Confidence Limits 



Let p be the likelihood that a data point is a signal. For N measurements the probability 
to observe k signals is given by the binomial coefficient 

MM= ( N k )p k( i N ~ k = k]{ N-k)\ pkqN ~ k with q = p ~ l - (3) 

The probability to observe k > N s signals is given by 

N 

Pk>N s (p)= E MM (4) 

k=N s 

and the probability to observe k < N s signals by 

N 

p k<N s (p) = EM^p) • ( 5 ) 

fc=0 

For N = 26 and N s = 10 the functional forms of Pk>N s (p) and Pk<N s (p) are depicted in 
figure [I]. 

Assume that N s signals are found in iV measurements and that a probability Q c < 0.5 
(typical values are Q c = 0.16 or Q c = 0.025) is given. We can solve equation (f|) for 
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Figure 1: The probability functions Pk>N s (p) to observe k > N s signals in events (£|) and 
Pk>N s (p) to observe k < N s signals in N events (|5[) are depicted for = 26 and N s = 10. 
Symmetric 68% confidence bounds ala Clopper-Pearson are also indicated. 



3 



Pk>N a (P-) — Q c and p_ is a lower bound on the true signal probability p ex act ; such that the 
likelihood to find k > N s signals in A" measurement is smaller than Q c for every p ex act < P-- 
Correspondingly, we can solve equation (^) for Pk<N s (p+) = Q c and p+ is an upper bound 
on the true signal probability p ex act> such that the likelihood to find k < N s signals in N 
measurement is smaller than Q c for every p ex act > P+- Together, this combines into the 
Clopper-Pearson bounds: The probability to find the true signal probability in the range 

P- < JWct < P+ is larger or equal to P c = 1 — 2Q C . (6) 

In more details the meaning of the inequality is discussed in [|J. For P c = 0.68 (Q c = 0.16) 
the p^ values are indicated in figure [1} Approximately, this range corresponds to the 
confidence of a la error bar. Similarly the confidence range corresponding to a 2a error 
bar, etc., can be found. 

3 Confidence Limits from Data Cuts 

We are interested in the situation where signal and background data can no longer be 
distinguished unambiguously. Instead, a neural network or other device yields statistical 
information by tagging signals with efficiency P s and background data with probability Pb, 
as discussed in the introduction. 

Applying the cuts to all A^ data results in N Y tagged data (0 < N Y < N), composed 
of A^ y = Nj + A^, where Nj is the number of tagged signals and N b Y is the number of 
tagged background data. Of course, the values for Nj and N b Y are not known. Our task is 
to determine confidence limits for the signal probability p from the sole knowledge of N Y . 
We proceed by writing down the probability density of N Y for given p and, subsequently, 
generalizing the Clopper-Pearson method. 

First, assume fixed N s . The probability densities of Nj and N Y are binomial and thus 
the probability density for A^ Y is given by the convolution 

P Y (N Y \N S )= £ b Ns (N?,P s )b Nb (N b Y ,P b ), N b = N-N s . (7) 

N Y +N Y =N Y 

Proof: For a signal event the probability to be tagged is P s , so bN s (Nj , P s ) is the probability 
to tag Nj out of the A^ s signals. Similarly, the probability for a background event to 
become tagged is P b and b]y b (N Y , P b ) is the probability to tag A^ of the N b = N — N s 
background events. As these two probabilities are independent, the likelihood that precisely 
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Ng of the signals and N b of the background events are tagged becomes the product 
bN s (Nj , P s ) bN b (N Y ,P b ). Summing over all possibilities which add up to Nj + N Y = N Y 
gives the result. 

Summing over N s in (|7]) removes the constraint of fixed N s and, with N, p fixed, the 
probability to tag k events becomes 

N 

b Y N (Kp)= E b N (N s ,p)P Y (k\N s ) . (8) 

N s =0 

Fourier transformation of the convolution (|7]) allows for an efficient numerical calculation of 
the P(k\N s ) coefficients. In analogy with equations (f|) and (j|) we find the probabilities to 

tag k > N Y and k < N Y events to be 

N N Y 

PI>nAp)= E bY N^P) and Pl< N y{p) = Y. hY N^P) ■ ( 9 ) 

k=N Y k=0 

For = 35, N Y = 12, P s = 0.8 and P b = 0.05 the functions § P%> n y{p) and P^ n y(p) are 
depicted in figure 0. The 68% confidence range (|B|) is also indicated in figure 0, where the 
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Figure 2: The probability functions Pk>N Y (p) to find k > N Y tags in iV events and P Y <NY (p) 
to find k < N Y tags in N events <g) are depicted for N = 35, N Y = 12, P s = 0.8 and 
Pb = 0.05. Symmetric 68% confidence bounds, found for p_ = 0.274 and p + = 0.521, are 
also indicated. 
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bound values p± are now defined as solutions of the equations 



Q c = Pk> N r(p-) and g c = PU NY { P+ ) ■ (io) 

The range [p_,p+], obtained with Q c = 0.16, guarantees the standard one error bar 
confidence probability of 68% for every true signal probability p ex act- For almost all values 
the actual confidence will be better. However, the bounds cannot be improved without 
violating the requested confidence probability for the case that p exa ct happens to agree with 
either p_ or p + . In the same way, bounds calculated with Q c = 0.023 ensure the standard 
two error bar confidence level of 95.4% or better, and so on. 

3.1 Data Sets with Few Signals 

As outlined, data sets with few signals are of of particular interest in high energy physics. 
Let us replace N Y = 12 of figure |2] by N Y = 3. The resulting graph is depicted in figure ^. 
From the P Y >NY {p) curve we read off the finite probability Pq = 0.254 for the likelihood that 
the true signal probability p exa ct = generates k > N y tags. Due to this probability the 
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Figure 3: The same functions as in figure |2] are depicted, but for N Y = 3 instead of N Y = 12 
tags. The lower 68% confidence bound does not exist anymore, instead the likelihood for 
Pexact = has become Pq = 0.254. The upper 68% confidence bound is found at p + = 0.149. 
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lower 68% confidence bound p_ disappears, whereas the upper p + bound does still exist. In 
passing let us note that for the data of figure ^| we have P = 0.69 10~ 7 , i.e. there p ex act = 
is ruled out on the 5a level. 

3.2 Signal Probability Distributions 

To avoid frequentist objections, the cumulative signal distribution function F(p) is, in the 
opinion of the author, best defined as the expectation of the researcher to find the true signal 
probability p ex act in the range < p ex act < P- Our approach allows to estimate upper and 
lower bounds for the cumulative signal distribution function 

rv 

F(p) = / dp f(p') where f(p) is the signal probability density. (11) 
Equation (|10|) implies 

N N 

F 1 (p) = l-P^ N r(p)= £ b Y N (k,p) < F(p) < F 2 (p) = P£> NY (P) = E b Y (k,p) 

k=N Y +l k=N Y 

(12) 
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Figure 4: Upper and lower bounds, F 2 (p) and Fi(p), for the cumulative signal distribution 
function F(p). The values used for N, N Y , P s and Pb are identical with those of figure |[ 
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and note that F(0) = F 2 (0). Figure ^ shows the result for the same data which were used in 
figure |2|. Any reasonable Bayesian estimates (which involves additional a-priori assumptions) 
should give a function F(p) which is sandwiched between Fi(p) and F 2 {p). 
It is instructive to define peaked distribution functions J7J by 

ppeaked/ x _ ( Fi(jP) for F^p) < 0.5 and , , 

1 [P> 1 1 - Fi{p) for Fi(p) > 0.5, (i = 1,2) . 1 6> 

Using the same data as in figure |], F 1 peaked (p) and i ? 2 >eaked (p) are depicted in figure [| 
The advantages of using peaked distribution functions instead of conventional cumulative 
distribution functions are: 

1. The ordinate becomes enlarged by a factor of two. 

2. The estimated medians are located at the peaks and the probability content of the 
distribution is instructively displayed. 

3. Bounds like those of figure [l] are easily read off. 
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Figure 5: Peaked distribution functions F 2 P ed (p) and ^(p) as defined in equation (|T 
The values used for N, N Y , P s and Pb are identical with those of figures ^| and |4j. 
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The probability densities corresponding to the cumulative distribution functions (O) are 
the derivatives of the Fi(p) with respect to p 

m = dJ j^r (i = 1 ' 2) - (14) 

Their numerical calculation is straightforward when analytical expressions for the derivatives 
of the binomial coefficients in equation (||) are used. Figure || exhibits the results for f\ (p) 
and fz(p) corresponding to F\{p) and F 2 {p) of figure fl[ At p = the probability densities 
have 5-function contributions 

/<(p) = F l (0)S(p) + ... , (2 = 1,2) . (15) 

In case of figure |6| the Fi(0) coefficients are practically zero. However, for the probability 
densities corresponding to figure [3] there would be a substantial contribution: F 1 (0) = 0.096 
and F 2 (0) = 0.254 in that case. 

3.3 The Fortran Code 

The Fortran code which produces the illustrations of this paper is available on the web. 
Start at the author's homepage || www.hep.fsu.edu/~berg and follow the research and 




0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 

P 

Figure 6: Probability densities fi(p) corresponding (|I4]) to the cumulative distribution 
function of figure f|. The values used for N, N Y , P s and P& are identical with those of 
figure £|. 
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from there the Clopper-Pearson hyperlink. Load down all files of the Fortran Programs 
subdirectory into an empty directory. Any standard Fortran 77 compiler should then be 
able to compile the cpl.f, cp2.f and cp3.f programs. Be aware that the program files 
include (via include Fortran statements) some of the other files you downloaded. Running 
one of the programs produces a data files with the name of that program and a . d extension. 
Subsequently, gnuplot users can produce the graphical presentations of this paper by using 
the * . pit driver files, as listed in the following. 

program generates file use with gnuplot driver 

cpl.f cpl.d cpl.plt 

cp2.f cp2.d cp2.plt, cp4.plt, cp5.plt, cp6.plt 

cp3.f cp3.d cp3.plt 

Here, the gnuplot file number corresponds to the figure number of this paper. To get 
encapsulated postscript files, the comment signs in front of the first two rows of each gnuplot 
file have to be eliminated. 



4 Conclusions 

We have calculated confidence limits, and corresponding limits of the entire cumulative 
distribution function, for an unknown true signal likelihood p cxa _ ct - The only input used are 
the efficiencies P s for tagging signals, the probabilities P^ for tagging background events, the 
number N Y of tagged data and the total number of data N. In particular, the method allows 
to deal with the situation where only few signals occur and yields then a finite probability for 
the likelihood that k > N Y tags are observed if the true signal probability is p cxa ct = 0. In real 
life the probabilities P s and are most likely estimators by themselves, i.e. quantities with 
error bars. This causes no major problem, one just has to apply our confidence calculations 
to an appropriate sample and to average over the results. 
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